As is known in the art, packet switching networks typically include a plurality of interconnected internal fabric switching units. Nodes are connected to the network through external network end point controllers. Both the internal network switching units and end points are herein sometimes collectively referred to herein as information packet controllers.
There are typically two types of information packet controllers: a push-only type usually associated with unidirectional messaging, wherein packets arrive at a target information packet controllers (herein sometimes referred to as a target) in the order they are generated at a source information packet controllers (herein sometimes referred to as a source); and a request/response hardware based request/response protocol) type where a source initiates a request for service in the form of an associated response packet from the target, said response packets bypassing request packets. One such example of such an associated request/response type is a load/store protocol in which a load request is responded to by the target with the requested data as a response with a data payload, and a store is responded to by the target with a receipt response.
With the push only type, while all packets are transmitted to the target, some applications require that certain classes of packets be transmitted with a higher quality of service than other classes of packets. Push only information packet controllers support one or more classes of quality of service, as an example guaranteeing a minimum bandwidth allocation. Typically, push only information packet controllers transmit packets through network information packet controllers in the order in which they arrive, otherwise known as First-In, First-Out (FIFO) order within each class of quality of service. Since there is no associated hardware related response required there is no issue with hardware induced deadlock scenarios.
With the request/response type, requests are sent to the target from the source and priority given to the responses to bypass requests being transmitted through the network. This allows the delivery of anticipated responses to free up resources within the system needed for processing requests, thereby avoiding system failure due to deadlock.
However, since the higher priority packets are sent first, some low priority packets may be starved, i.e., may be not transmitted to, and hence not serviced by, the target. Typically, request/response capable networks will employ a scheme in which the packet contains a priority tag, often providing multiple levels of priority, while stipulating that the response associated with a given request must be of a higher priority than that request. Multiple levels of priority available for request packets offers a form of quality of service usually associated with delivery time. Packets are usually issued and promoted through the network in priority order, hence avoiding the aforementioned deadlock scenarios, but not supporting other quality of service considerations.
However, since the higher priority packets are sent first, some low priority packets may be starved, i.e., may not be transmitted to, and hence not serviced by, the target for extended periods of time. In the extreme, the presence of higher priority packets from one or more information packet controllers can cause lower priority packets from another information packet controller to never be transmitted.
As is also known in the art, there are two types of packet flow control: a receiver based flow control and an initiator based flow control. This flow control is employed at each connection, sometimes referred to as a link or bus, between information packet controllers within the network. Each information packet controller has one initiator and one receiver associated with each link to an attached other information packet controller. It is first noted that the receiver has only a limited number of locations for temporary storage of packet being transmitted. In the initiator based flow control, number of available locations is indented by a number of credits, communicated by the issuance by the receiver of Credit flow control packets.
With target based flow control, the initiator speculatively sends a packet to the receiver. The receiver advises the initiator to retry if the receiver is not able or does not wish to receive the packet sent to it by the initiator i.e., if the receiver is not ready or does not want the packet sent to it by the initiator the receiver replies by sending the initiator a Retry flow control packet. The retry may be sent by the receiver if the priority indication appended to the packet sent by the initiator is a priority lower than that desired by the receiver in view of the available storage at the receiver.
With initiator, based flow control the receiver informs the initiator as to the number of available storage locations, i.e., the number of credits. The initiator then determines the packet and number of packets to be sent to the receiver, reserving sufficient available locations to support sending higher priority packets.
As noted above, there are typically two types of information packet controllers: a push-only type and a request/response type. Push only type networks can leverage a scheme of promoting packets through the network based on the order of arrival, avoiding excessive service times while utilizing full bandwidth, meanwhile supporting different classes of service. With the request/response type, a prioritization is required. However since the higher priority packets are sent first, some low priority packets may be starved, i.e., may not be transmitted to, and hence not serviced by, the target. Note also that, unlike the push-only model where packets are promoted in arrival order independent of packet size, load/store packet sizes have a direct impact on the transmission performance characteristics of the network.
Request packets expecting a response data payload (e.g. load requests) are typically small in size, stipulating just enough information to locate or label the desired data and stipulate the data size to be transferred. The response packets with the requested data payload, in contrast, are of a size to satisfy the request, hence are typically much larger.
Request packets sending a data payload (e.g. store requests) are typically large in size to accommodate the data payload. In a guaranteed delivery network, typically request packets sending a data payload will be responded to by the target hardware by the sending of a receipt acknowledgement packet, with such associated hardware acknowledge response packet being very small.
Consider, for example, two interconnected of the information packet controllers wherein one information packet controller, (controller 1), has attached two controller sources (sourceA1 and sourceB1) and two target memory controllers (targetA1 and targetB1), and the other packet controller (controller 2), also has attached two packet controller sources (sourceA2 and sourceB2) and two target memory controllers (targetA2 and targetB2) in accordance with the following table:
Controller 1Controller 2sourceA1sourceA2sourceB1sourceB2targetA1targetA2targetB1targetB2
In this example, each source is capable of issuing multiple outstanding load or store requests (segments of larger transfers) at information packet controller link interconnect rates. The two sources (sourceA1 and sourceB1) on packet controller 1 start first, queuing up (small) priority 0 load requests in a packet controller-to-packet controller input buffer in packet controller 1, which then streams to packet controller 2's input buffer, and hence to packet controller 2's memory controllers (targetA2 and targetB2). Memory controllers targetA2 and targetB2 then establish a stream of (large) priority 1 data payload packets back through a packet controller-to-packet controller input buffer in packet controller 2, and hence through packet controller 1's input buffer and on to sourceA1 and sourceA2. The limit in this case will be the available packet controller to packet controller link bandwidth. Assume that the responses in the packet controller-to-packet controller input buffer in packet controller 1 are being sent to the two requesters in parallel, such that the packet controller-to-packet controller input buffer in packet controller 1 never fills up. Then the two requesters on packet controller 2 start issuing (small) priority 0 load request packets. These two request streams are now competing for the packet controller 2 output port. According to network protocols as exemplified by the Serial Rapid I/O Specification which impose strict priority based arbitration, the packet controller 2 output port will always service the highest priority requester. There will always be at least one memory controller with a priority 1 packet waiting. As a consequence, the requesters on packet controller 2 will never get any priority 0 requests out the packet controller 2 output port onto the packet controller-to-packet controller link, even though the packet controller to packet controller input buffer in packet controller 1 could accept them. Note that although the link from packet controller 2 to packet controller 1 is fully utilized, the throughput on the link from packet controller 1 to packet controller 2 only uses a fraction of the available bandwidth proportional to the size of the request packets to the size of the response packets. The disadvantage of always promoting the highest priority packets available is that it lowers throughput and causes starvation (i.e., lack of service) of lower priority packets resulting in either lack of service or excessive latencies of services.